Haptic trackball device

ABSTRACT

A low-cost haptic feedback trackball device for providing haptic feedback to a user for enhancing interactions in a graphical environment provided by a computer. The trackball device includes a sensor device that detects the movement of a sphere in two rotary degrees of freedom. An actuator applies a force preferably along a z-axis perpendicular to the plane of the surface supporting the device, where the force is transmitted through the housing to the user. The output force is correlated with interaction of a controlled graphical object, such as a cursor, with other graphical objects in the displayed graphical environment. Preferably, at least one compliant element is provided between a portion of the housing contacted by the user and the support surface, where the compliant element amplifies the force output from the actuator by allowing the contacted portion of the housing to move with respect to the support surface. The force can be an inertial force, contact force, or a combination of forces that provide tactile sensations to the user.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation-in-part of co-pending parentpatent applications:

[0002] Application Ser. No. 09/103,281, filed Jun. 23, 1998 on behalf ofLouis Rosenberg, entitled, “Low Cost Force Feedback Device with Actuatorfor Non-Primary Axis,”

[0003] Application Ser. No. 09/253,132, filed Feb. 18, 1999 on behalf ofLouis Rosenberg, entitled, “Low Cost Force Feedback Pointing Device,”and

[0004] Application Ser. No. 09/456,887, filed Dec. 7, 1999 on behalf ofLouis Rosenberg, entitled, “Tactile Mouse Device,”

[0005] all assigned to the assignee of this present application, and allof which are incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION

[0006] The present invention relates generally to interface devices forallowing humans to interface with computer systems, and moreparticularly to computer interface devices that allow the user toprovide input to computer systems and allow computer systems to providehaptic feedback to the user.

[0007] A user can interact with an environment displayed by a computerto perform functions and tasks on the computer, such as playing a game,experiencing a simulation or virtual reality environment, using acomputer aided design system, operating a graphical user interface(GUI), etc. Common human-computer interface devices used for suchinteraction include a mouse, joystick, trackball, steering wheel,stylus, tablet, pressure-sensitive sphere, or the like, that isconnected to the computer system controlling the displayed environment.Typically, the computer updates the environment in response to the usersmanipulation of a physical manipulandum such as a joystick handle ormouse, and provides visual and audio feedback to the user utilizing thedisplay screen and audio speakers. The computer senses the user'smanipulation of the user object through sensors provided on theinterface device that send locative signals to the computer. Forexample, the computer displays a cursor or other graphical object in agraphical environment, where the location of the cursor is responsive tothe motion of the user object.

[0008] In some interface devices, force feedback or tactile feedback isalso provided to the user, more generally known herein as “hapticfeedback.” These types of interface devices can provide physicalsensations which are felt by the user manipulating a user manipulandumof the interface device. One or more motors or other actuators arecoupled to the joystick or mouse and are connected to the controllingcomputer system. The computer system controls forces on the joystick ormouse in conjunction and coordinated with displayed events andinteractions by sending control signals or commands to the actuators.The computer system can thus convey physical force sensations to theuser in conjunction with other supplied feedback as the user is graspingor contacting the interface device or manipulatable object of theinterface device. For example, when the user moves the manipulatableobject and causes a displayed cursor to interact with a differentdisplayed graphical object, the computer can issue a command that causesthe actuator to output a force on the physical object, conveying a feelsensation to the user.

[0009] One problem with current force feedback controllers in the homeconsumer market is the high manufacturing cost of such devices, whichmakes the devices expensive for the consumer. A large part of thismanufacturing expense is due to the inclusion of multiple actuators andcorresponding control electronics in the force feedback device. Inaddition, high quality mechanical and force transmission components suchas linkages and bearings must be provided to accurately transmit forcesfrom the actuators to the user manipulandum and to allow accuratesensing of the motion of the user object. These components are complexand require greater precision in their manufacture than many of theother components in an interface device, and thus further add to thecost of the device. A need therefore exists for a haptic device that islower in cost to manufacture yet offers the user haptic feedback toenhance the interaction with computer applications.

SUMMARY OF THE INVENTION

[0010] The present invention is directed to a low-cost haptic feedbacktrackball device connected to a computer system, the trackball devicehaving a simple actuator for low cost force feedback for enhancinginteractions and manipulations in a displayed graphical environment.

[0011] More specifically, the present invention relates to a hapticfeedback trackball device that is coupled to a host computer whichimplements a host application program. The device includes a housingthat is physically contacted by said user, the housing resting on asupport surface. A sphere is positioned in the housing, the sphere beingrotatable in two rotary degrees of freedom. A sensor device detects themovement of the sphere in the rotary degrees of freedom and outputssensor signals representative of the movement. An actuator applies aforce to the housing approximately along an axis that is substantiallyperpendicular to the support surface, where the force is transmitted tothe user contacting the housing. The force is preferably correlated witha graphical representation displayed by the host computer, where aposition of the sphere in the rotary degrees of freedom corresponds witha position of a cursor displayed in the graphical representation.

[0012] Preferably, at least one compliant element is provided between aportion of the housing contacted by the user and the support surface,where the compliant element amplifies the force output from the actuatorby allowing the contacted portion of the housing to move with respect tothe support surface. For example, the compliant element can be one ormore feet provided on the underside of the housing and made of acompliant material such as rubber or foam. Or, the compliant element canbe a compliant coupling provided between the contacted portion of thehousing and a non-contacted portion of the housing.

[0013] In some embodiments, the force is an inertial force that isoutput approximately along the axis that is substantially perpendicularto the support surface, where the actuator outputs the inertial force tothe housing by moving an inertial mass. The actuator can be coupled to aflexure that provides a centering spring bias to the inertial mass. Theinertial force can be a pulse, vibration or texture correlated with theinteraction of a user-controlled cursor with a graphical objectdisplayed in a graphical user interface. For example, the pulse can beoutput when the cursor moves between menu items in a displayed graphicalmenu. In other embodiments, the force is a contact force that isprovided by driving a moving element that contacts the user. The movingelement can be a cover portion of the housing that is movably coupled toa base portion of the housing. Alternatively, the moving element can bea button that also provides button input to the host computer. Someembodiments may include a second actuator, such as a passive brake, foroutputting a force on the sphere in its degrees of freedom. A method forproviding haptic feedback similarly includes detecting the motion of asphere of the trackball device, receiving information from the hostcomputer indicating that a tactile sensation is to be output, andoutputting a force on the housing of the trackball device approximatelyalong an axis perpendicular to a support surface.

[0014] The present invention advantageously provides a haptic feedbacktrackball device that is significantly lower in cost than other types ofhaptic feedback devices and is thus quite suitable for home consumerapplications. A single actuator can be provided that applies a force ina particular degree. of freedom, such as the Z-axis perpendicular to thesupport surface, and compliance is provided between surface and usercontact. This allows more compelling forces to be experienced by theuser, and also enhances the user's experience of a third dimensionrelative to the surface plane. Furthermore, the actuator of the presentinvention can provide a variety of different types of force sensationsto enhance the user's interfacing and experience with a computerapplication.

[0015] These and other advantages. of the present invention will becomeapparent to those skilled in the art upon a reading of the followingspecification of the invention and a study of the several figures of thedrawing.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016]FIG. 1 is a perspective view of system including a haptictrackball device of the present invention connected to a host computer;

[0017]FIG. 2 is a side cross sectional view of the trackball device ofFIG. 1 providing inertial forces;

[0018]FIG. 3 is a perspective view of one embodiment of an actuatorassembly suitable for use with the present invention;

[0019]FIG. 4 is a side cross sectional view of the trackball device ofFIG. 1 providing contact forces;

[0020]FIG. 5 is a block diagram of the haptic device and host computerof the present invention; and

[0021]FIG. 6 is a diagrammatic view of a display screen showinggraphical objects associated with force sensations output using thehaptic device of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0022]FIG. 1 is a perspective view of a haptic feedback interface system10 of the present invention capable of providing input to a hostcomputer based on the user's manipulation of a trackball and capable ofproviding haptic feedback to the user of the interface system based onevents occurring in a program implemented by the host computer. System10 includes a trackball device 12 and a host computer 14. It should benoted that the term “trackball” as used herein, indicates any device inwhich a spherical object can be rotated by the user to provide input tothe host computer.

[0023] Trackball device 12 includes a housing 13 and a sphere or ball15. Sphere 15 is contacted by a user's finger and/or palm and is rotatedin any direction, e.g. in two degrees of freedom. In the embodimentshown, the user preferably rests his or her palm on the housing 13 whenmoving the sphere 15 to provide a hand rest. In other embodiments, thetrackball device can provide a large sphere 15 and a small housing 13with buttons, or can provide a sphere that can be contacted in multipleareas, such as with a thumb and a forefinger on both sides of thesphere.

[0024] The sphere 15 can be rotated in the two degrees of freedom toprovide input to the host computer 14. For example, a user can movesphere 15 to provide two-dimensional input to a computer system tocorrespondingly move a computer generated graphical object, such as acursor or other image, in a graphical environment provided by computer14 or to control a virtual character, vehicle, or other entity in a gameor simulation. In addition, trackball device 12 preferably includes oneor more buttons 16 a and 16 b to allow the user to provide additionalcommands to the computer system, and may also include additionalbuttons.

[0025] Trackball device 12 preferably includes an actuator 18 which isoperative to produce forces on the trackball device 12. This operationis described in greater detail below with reference to FIG. 2.

[0026] Trackball device 12 rests on a ground surface 22 such as atabletop or other reference surface. A user contacts the sphere 15 andhousing 13 while using the device 12. Since the sphere 15 can move inboth directions without moving the housing 13, the housing typicallyremains stationary with respect to the surface 22. Sensor mechanismsused to detect and measure sphere 15 rotation are described below withreference to FIG. 2.

[0027] Trackball device 12 is preferably a relative device, in which thedevice 12 reports a change in position to the host computer and the hostcontrols a graphical object, adjusts a value, etc., based on the changein position. Thus, the sphere 15 can be rotated indefinitely in anydirection. In other embodiments, the trackball device 12 may beimplemented as an absolute device, in which the sphere 15 has anabsolute position in a workspace and the absolute position is reportedto the host computer. In one absolute embodiment, stops can be placed inthe workspace of the sphere 15 to prevent the sphere from moving outsidethe bounded workspace.

[0028] Trackball device 12 is coupled to the computer 14 by a bus 20,which communicates signals between device 12 and computer 14 and mayalso, in some preferred embodiments, provide power to the trackballdevice 12. Components such as an actuator (described below) requirepower that can be supplied from through the bus 20 if the bus is, forexample, a USB or Firewire bus. In other embodiments, signals can besent between trackball device 12 and computer 14 by wirelesstransmission/reception. In some embodiments, the power for the actuatorcan be supplemented or solely supplied by a power storage deviceprovided on the device 12, such as a capacitor or one or more batteries.Some embodiments of such are disclosed in U.S. Pat. No. 5,691,898,incorporated herein by reference.

[0029] Host computer 14 is preferably a personal computer orworkstation, such as a PC compatible computer or Macintosh personalcomputer, or a Sun or Silicon Graphics workstation. For example, thecomputer 14 can operate under the Windows™, MacOS, Unix, or MS-DOSoperating system. Alternatively, host computer system 14 can be one of avariety of home video game console systems commonly connected to atelevision set or other display, such as systems available fromNintendo, Sega, or Sony. In other embodiments, host computer system 14can be a “set top box” which can be used, for example, to provideinteractive television functions to users, a “network-” or“internet-computer” which allows users to interact with a local orglobal network using standard connections and protocols such as used forthe Internet and World Wide Web, or other appliance or device allowingthe user to provide two-dimensional (or greater) input for selection orcontrol. Host computer preferably includes a host microprocessor, randomaccess memory (RAM), read only memory (ROM), input/output (I/O)circuitry, and other components of computers well-known to those skilledin the art.

[0030] Host computer 14 preferably implements a host application programwith which a user is interacting via trackball device 12 and otherperipherals, if appropriate, and which may include force feedbackfunctionality. For example, the host application program can be a videogame, word processor or spreadsheet, Web page or browser that implementsHTML or VRML instructions, scientific analysis program, virtual realitytraining program or application, or other application program thatutilizes input of device 12 and outputs force feedback commands to thedevice 12. Herein, for simplicity, operating systems such as Windows™,MS-DOS, MacOS, Linux, Be, etc. are also referred to as “applicationprograms.” In one preferred embodiment, an application program utilizesa graphical user interface (GUI) to present options to a user andreceive input from the user. Herein, computer 14 may be referred asproviding a “graphical environment”, which can be a graphical userinterface, game, simulation, or other visual environment. The computerdisplays “graphical objects” or “computer objects,” which are notphysical objects, but are logical software unit collections of dataand/or procedures that may be displayed as images by computer 14 ondisplay screen 26, as is well known to those skilled in the art. Adisplayed cursor or a simulated cockpit of an aircraft might beconsidered a graphical object. The host application program checks forinput signals received from the electronics and sensors of trackballdevice 12, and outputs force values and/or commands to be converted intoforces output for trackball device 12. Suitable software drivers whichinterface such simulation software with computer input/output (I/O)devices are available from Immersion Corporation of San Jose, Calif.

[0031] Display device 26 can be included in host computer 14 and can bea standard display screen (LCD, CRT, flat panel, etc.), 3-D goggles, orany other visual output device. Typically, the host application providesimages to be displayed on display device 26 and/or other feedback, suchas auditory signals. For example, display screen 26 can display imagesfrom a GUI.

[0032] As shown in FIG. 1, the host computer may have its own “hostframe” 28 which is displayed on the display screen 26. In contrast, thedevice 12 has its own workspace or “local frame” in which the sphere 15is moved. In a position control paradigm, the position (or change inposition) of a user-controlled graphical object, such as a cursor, inhost frame 28 corresponds to a position (or change in position) of thesphere 15 in the local frame. The offset between the object in the hostframe and the object in the local frame can be changed by the user byindexing, i.e., moving the sphere 15 while no change in input isprovided to the host computer. Indexing is typically not needed for atrackball since the workspace of the sphere 15 is infinite.

[0033] In alternative embodiments, the device 12 can be a differentinterface or control device. For example, a hand-held remote controldevice used to select functions of a television, video cassetterecorder, sound stereo, internet or network computer (e.g., Web-TV™),mouse device, or a gamepad controller for video games or computer games,can include a sphere 15 for input and can be used with the hapticfeedback components described herein. Handheld devices can still benefitfrom the directed inertial sensations described herein which, forexample, can be output perpendicularly from the device's top surface. Inyet other embodiments, the actuator 18 (and all the variations describedherein) can be positioned within a handle of a large joystick, and soprovide tactile sensations such as pulses and vibrations to the usergrasping the joystick handle. Similarly, the actuator embodimentsmentioned herein can be placed in the grasped steering wheel of a wheelcontroller. The actuator assembly can be scaled to the desired size toprovide haptic sensations appropriate for the size and mass of thecontroller device.

[0034]FIG. 2 is a side cross-sectional view of the trackball device 12of FIG. 1. Trackball device 12 includes one or more actuators 18 forimparting haptic feedback such as tactile sensations to the user of thedevice 12. The actuator outputs forces on the device 12 which the useris able to feel.

[0035] Trackball device 12 includes a housing 13, a sensing system 40,and an actuator 18. Housing 13 can be provided in a variety of shapes toallow the user to manipulate the sphere 15 and buttons 16. Sensingsystem 40 detects the position of the sphere in its two rotary degreesof freedom. In some embodiments, sensing system 40 can includecylindrical rollers 52 which are coupled to sensors 54, such as opticalencoders, for detecting the motion of the sphere 15. Each roller 52 isfrictionally coupled to the sphere 15 and rotates when the sphere 15rotates, and the associated sensor 54 detects the rotation of theroller. A roller and sensor can be used for each of the degrees offreedom of the sphere 15, e.g. the x-direction and y-direction.

[0036] Other types of mechanisms and/or electronics for detecting motionof the sphere 15 can be used in other embodiments. For example, sometrackball devices employ non-contact optical emitters and detectors tosense motion of the sphere 15. In some of these embodiments, the motionof a surface or pattern on the sphere, such as dots or bars, isdetected. Such optical sensing methods can be used in the presentinvention. Other types of sensors can also be used, such as magneticsensors, analog potentiometers, etc.

[0037] An actuator 18 is coupled to the housing 13 to provide hapticfeedback to the user. The haptic feedback can generally be provided intwo forms: inertial forces and contact forces. Inertial forces areprovided by moving an inertial mass, which causes forces on the housingfelt by the user. Contact forces are more direct forces applied to theuser, such as by moving an element of the housing which contacts theuser's hand.

[0038] A preferred embodiment creates inertial forces that are directedsubstantially in a particular degree of freedom, i.e. along a particularaxis. The inertial forces can be created, for example, using a highbandwidth linear actuator; preferred actuators include a linear movingvoice coil actuator and a linear moving-magnet actuator, which aresuitable for high bandwidth actuation. A traditional servo motor used ina harmonic drive configuration can also be a suitable high bandwidthactuator. This embodiment allows for high fidelity control of forcesensations in both the frequency and magnitude domains. This also allowsthe forces to be directed along a desired axis and allows for crisptactile sensations that can be independently modulated in magnitude andfrequency.

[0039] The preferred direction for the output forces is along theZ-axis. Since the tactile sensations are directed in a third degree offreedom relative to the two-dimensional planar surface and displayscreen, jolts or pulses output along the Z axis feel much more likethree-dimensional bumps or divots to the user, increasing the realism ofthe tactile sensations and creating a more compelling interaction. Forexample, an upwardly-directed pulse that is output when the cursor ismoved over a window border creates the illusion that the sphere or wholedevice 12 is moving “over” a bump at the window border.

[0040] In a first inertial force feedback embodiment, actuator 18 ispreferably a linear actuator having a stationary portion coupled to thedevice housing 13 (and thus stationary only with respect to the portionof the housing to which it is coupled), and a moving portion that moveslinearly approximately along the Z-axis. The stationary portion mayinclude a magnet and the moving portion can include a wire coil; orthese components can be reversed. An inertial mass can be coupled to thelinearly-moving portion of the actuator. The actuator 18 is operative tooscillate the inertial mass quickly parallel to the Z axis. Thus, forcesproduced by the moving mass are transmitted to the housing through thestationary portion of the actuator 18 and felt by the user as tactilesensations. These forces are substantially directed along the Z axis andthus do not generally interfere with the user moving the sphere 15 aboutthe x- and y-axes.

[0041] Actuator 18 can be a linear voice coil actuator as described incopending patent application Ser. No. 09/253,132, which is incorporatedherein by reference. In other embodiments, the stationary portion can bethe coil and the moving portion can be the magnet. Actuator 18 can beother types of actuators in other embodiments. For example, a rotaryactuator can be used having rotational force output that is converted tolinear force output. A pager motor or other actuator having a rotatingshaft, a solenoid having a vertically-moving portion, a linear voicemagnet, DC current controlled linear motor, a linear stepper motorcontrolled with pulse width modulation of an applied voltage, apneumatic/hydraulic actuator, a torquer (motor with limited angularrange), a piezo-electric actuator, etc., can be used. A rotary actuatorcan be used to output a torque in a rotary degree of freedom on a shaft,which is converted to linear force and motion through a transmission, asis well known to those skilled in the art.

[0042] The actuator 18 can be placed in a variety of positions withinthe housing 13. For example, one preferred embodiment places theactuator on the bottom portion of the housing, as close to the center ofthe device 12 along both the X and Y axes as possible. In otherembodiments, the actuator 18 can be positioned centered along one axisbut off-center along the other axis to accommodate other electronic andmechanical components in the device, e.g. near the front or back of thehousing 13. In yet other embodiments, the actuator 18 can be connectedto a side or top portion of the housing 13 rather than the bottomportion, although it is preferred that the actuator be oriented tooutput forces approximately along the Z-axis (and thus the top may bepreferable to the side). A variety of tactile sensations can be outputto the user, many of which are described in greater detail below withrespect to FIG. 6.

[0043] The magnitude of forces that can be output with respect to aninertial ground are not as high as can be output with respect to anearth ground. The larger the inertial mass, the larger the forces thatcan be output, so the theoretical limit of force magnitude is very high.Since the trackball device housing 13 does not need to be moved by theuser to operate the device, the inertial mass can be made fairly largeto provide higher magnitude forces. Size may be a constraint, however,in some devices.

[0044] In addition, a high bandwidth actuator can be used to compensatefor lower-magnitude forces, i.e., an actuator that can output abruptchanges in force magnitude level. Since the human hand is more sensitiveto changes in force level than to absolute force levels, a highbandwidth actuator used to convey low level forces produced with respectto an inertial ground can be quite effective in producing compellingtactile sensations.

[0045] An additional challenge of applying a compelling tactilesensation to the housing 13 along the described Z axis is that thetrackball device 12 sits upon a table or other surface 22 and istherefore physically grounded along that Z axis. In other words, theforces applied by the actuator 18 along the Z axis, with respect to theinertial mass, are countered by the normal forces applied by the tablesurface upon the housing. One way to accommodate these countering forcesand to allow greater magnitude forces to be felt by the user is toprovide compliance between the surface 22 and a portion of the housingthat is contacted by the user. In a preferred embodiment, a flexible orsemi-flexible surface is provided between the housing 13 and the surface22. For example, a number of compliant feet 60 can be coupled to theunderside of the housing 13 to make contact with surface 22. The feet 60can be made out of a material such as rubber, foam, or the like.Preferably, the feet have a high compliance in the z-axis to allow thedesired magnitude of haptic sensations in the z-axis. Descriptions oftuning compliance to provide greater-magnitude forces are provided incopending application No. 60/157,206, incorporated herein by reference.In other embodiments, a whole layer of compliant material can bepositioned underneath or coupled to the underside of the housing 13.

[0046] In other embodiments, the desired compliance can be provided inother or additional elements of the device 12. For example, a coverportion of the device can be flexibly or moveably coupled to a baseportion of the device 12, where z-axis motion between these portionsmagnifies the haptic sensations. In one embodiment, the top half of thehousing 13 which the user contacts can be coupled to the bottom half bya rubber joint or other flexible layer or coupling. In otherembodiments, the contacted cover portion can be a smaller portion of thetop surface of the housing 13 which is compliant, e.g. a rubberdiaphragm. It should be noted that such a compliant cover portion is notdriven directly by the actuator (as is the case for the contact forcesof FIG. 4), but is provided to more efficiently transmit inertial forcesto the user.

[0047] Alternate embodiments include coupling the stationary portion ofthe actuator 18 to a portion of the housing 13 that is different fromthe base or bottom portion of the housing (e.g. the side of thehousing), and providing an amount of flex between the actuator-coupledportion of the housing and the base portion that is in contact with thesurface 22. For example, flexible hinges or connecting members cancouple the two portions. This can improve the transmissibility of thetactile sensations, leader to greater magnitude forces.

[0048] A different implementation that may be used for generatingtactile sensations is a motor (or other actuator) having a rotatingshaft, where an inertial mass is connected to the shaft at an off-centerpoint of the mass. The inertial mass is rotated around the motor shaftwith respect to the interface device at various speeds. This can createsinusoidal force signals at various frequencies depending upon thecurrent driven through the motor. One problem with such a methodology isslow response time because the spinning mass must accelerate anddecelerate over time to achieve the rotational velocity corresponding toa desired frequency output. Also, this implementation applies forces ina continually changing direction confined to the plane of rotation ofthe mass, which may provide a “wobble” sensation that can bedisconcerting to the user at slow frequencies.

[0049] Alternatively, directed inertial forces can be output along the Xand Y axes in the planar workspace of the device and can be compensatedfor to prevent or reduce interference with the user's control of thedevice. One method to compensate is to actively filter imparted jitterin that workspace, as disclosed in a pending patent application Ser. No.08/839,249, incorporated herein by reference; however, thisimplementation may add complexity and cost to the device 12. One problemin the present invention for outputting forces in the X and Y directionsis that the housing 13 is typically made stiff in those directions, suchthat forces will not be easily felt. For example, the rubber feet 60 canbe made compliant in the z-direction, but such compliance does notgreatly help to magnify forces output in the X- and/or Y- axes. Therubber feet 60 are typically stiff in the x-y plane to prevent thehousing 13 from wobbling when the user uses the device 12.

[0050] Buttons 16 can be selected by the user as a “command gesture”when the user wishes to input a command signal to the host computer 14.The user pushes a button 16 down (in the degree of freedom of the buttonapproximately along axis z) to provide a command to the computer. Thecommand signal, when received by the host computer, can manipulate thegraphical environment in a variety of ways. In one embodiment, anelectrical lead can be made to contact a sensing lead as with anymechanical switch to determine a simple on or off state of the button.An optical switch or other type of digital sensor can alternatively beprovided to detect a button press. In a different continuous-rangebutton embodiment, a sensor can be used to detect the precise positionof the button 16 in its range of motion (degree of freedom). In someembodiments, one or more of the buttons 16 can be provided with forcefeedback (in addition to the inertial tactile feedback from actuator18), as described in copending patent application Ser. No. 09/235,132.

[0051]FIG. 3 shows an example of an actuator assembly 80 that can beused in the present invention. In this embodiment, the actuator itselfis used as the inertial mass. An actuator assembly is used that includesa flexure for providing inertia forces and which includes an inherentspring bias that brings the inertial mass back to an origin positionwhen no forces are output on the mass.

[0052] Actuator assembly 80 includes a grounded flexure 68 and anactuator 66 coupled to the flexure 68. The flexure 68 is preferably asingle, unitary piece made of a material such as polypropylene plastic(“living hinge” material) or other flexible material. This type ofmaterial is durable and allows flexibility of the flex joints (hinges)in the flexure when one of the dimensions of the joint is made small,but is also rigid in the other dimensions, allowing structural integrityas well as flexibility depending on thickness. Some embodiments offlexures used in force feedback devices are described in Patent5,805,140 and patent application Ser. Nos. 09/376,649 and 60/______,entitled “Haptic Interface Device Providing Linear Tactile SensationsUsing A Rotary Actuator,” filed Dec. 21, 1999, all incorporated hereinby reference. Flexure 68 can be grounded to the housing 13, for example,at portion 81.

[0053] Actuator 66 is shown coupled to the flexure 68. The housing ofthe actuator is coupled to a receptacle portion 82 of the flexure 68which houses the actuator 66 as shown. Preferably, an amount of space isprovided above and below the actuator 66 and receptacle portion 82 toallow motion of the actuator 66 in the z-axis; thus, the receptacleportion 82 should not be coupled to ground since it moves to provide anapproximately linear motion, as explained below.

[0054] A rotating shaft 84 of the actuator is coupled to the flexure 68in a bore 85 of the flexure 68 and is rigidly coupled to a centralrotating member 90. The rotating shaft 84 of the actuator is rotatedabout an axis A which also rotates member 90 about axis A. Rotatingmember 90 is coupled to a first portion 92 a of an angled member 91 by aflex joint 94. The flex joint 94 preferably is made very thin in thedimension it is to flex, i.e. one of the x- or y-axis dimensions (they-axis dimension for the embodiment of FIG. 3), so that the flex joint94 will bend when the rotating portion 90 moves the first portion 92 aapproximately linearly. The first portion 92 a is coupled to thegrounded portion 100 of the flexure by a flex joint 98 and the firstportion 92 a is coupled to a second portion 92 b of the angled member byflex joint 102. The second portion 92 b, in turn, is coupled at itsother end to the receptacle portion 82 of the flexure by a flex joint104.

[0055] The angled member 91 that includes first portion 92 a and secondportion 92 b moves approximately linearly along the x-axis as shown byarrow 96. When the flexure is in its origin position (rest position),the portions 92 a and 92 b are preferably angled as shown with respectto their lengthwise axes. This allows the rotating member 90 to push orpull the angled member 91 along either direction as shown by arrow 96.This configuration allows forces output by the actuator to be magnifiedas they are transmitted to the moveable receptacle portion 82 and to themoving element of the interface device (inertial mass, cover portion,button, etc.). The actual force output depends on the angle of theopposing portions 92 a and 92 b with respect to each other's lengthwiseaxes (or with respect to the y-axis).

[0056] The actuator 66 is operated in only a fraction of its rotationalrange when driving the rotating member 90 in two directions, allowinghigh bandwidth operation and high frequencies of pulses or vibrations tobe output. The resulting motion of the angled member 91 compresses orstretches the flexure with respect to the grounded portion 81. Tochannel this compression or stretching into the desired z-axis motion, aflex joint 112 is provided in the flexure portion between the receptacleportion 82 and the grounded portion 100. Flex joint 112 is oriented toflex along the z-axis (i.e. provide rotation about an x-axis), unlikethe flex joints 94, 98, 102, and 104, which flex in the x-y plane(provide rotation about a z-axis). The flex joint 112 allows thereceptacle portion 82 (as well as the actuator 66, rotating member 90,and second portion 92 b) to move linearly in the z-axis in response tomotion of the portions 92 a and 92 b. In actuality, the receptacleportion 82 and actuator 66 move only approximately linearly, since theyhave a small arc to their travel; however, this arc is small enough tobe ignored for most practical purposes. Thus, when the rotational motionof the rotating member 90 causes the ends of the angled member 91 tomove further apart (direction 106 a), the receptacle portion flexes downabout flex joint 112 along the z-axis. Similarly, if the ends of angledmember 91 are made to move closer together (direction 106 b), thereceptacle 82 and actuator 66 move upwardly along the z-axis, in effectlifting the actuator 66 upward. A flex joint 110 is provided in thefirst portion 92 a of the angled member 91 to allow the flexure aboutflex joint 112 in the z-direction to more easily occur. The essentialelements of the schematic embodiment shown in FIG. 3 can be implementedwith a wide variety of components, including mechanical couplings suchas bearings, pin joints, etc.

[0057] By quickly changing the rotation direction of the actuator shaft84, the actuator/receptacle can be made to oscillate along the z-axisand create a vibration on the housing with the actuator 66 acting as aninertial mass. Preferably, enough space is provided above and below theactuator to allow its range of motion without impacting any surfaces orportions of the housing 13, since such impacts can degrade the qualityof the pulse, vibrations, and other haptic sensations output to theuser.

[0058] In addition, the flex joints included in flexure 68, such as flexjoint 112, act as spring members to provide a restoring force toward theorigin position (rest position) of the actuator 66 and receptacleportion 82. This centering spring bias reduces the work required by theactuator to move itself since the actuator output force need only bedeactivated once the actuator reaches a peak or valley position in itstravel. The spring bias brings the actuator back to its rest positionwithout requiring actuator force output. This system can be tuned sothat amplification of forces output by the actuator is performed at aefficient level, e.g. near the natural frequency of the system. Tuningsuch a harmonic system using an inertial force actuator and compliantsuspension of a moving mass is described in greater detail in copendingprovisional patent application Ser. No. 60/157,206, which isincorporated herein by reference. For example, in the flexure 68, thespring constants can be tuned by adjusting the thickness of the flexjoints 94, 102, 98, 104, 110, and/or 112 (in the dimension in which theyare thin). In some embodiments, additional springs can be added toprovide additional centering forces if desired, e.g. mechanical springssuch as leaf springs.

[0059] The flexure 68 is advantageous in the present invention becauseit has an extremely low cost and ease of manufacturability, yet allowshigh-bandwidth forces to be transmitted as inertial forces. Since theflexure 68 is a unitary member, it can be manufactured from a singlemold, eliminating significant assembly time and cost. Furthermore, it isrigid enough to provide strong vibrations with respect to the housingand to provide significant durability. In addition, the flexure providesclose to zero backlash and does not wear out substantially over time,providing a long life to the product.

[0060] Providing the actuator 66 as the inertial mass that is driven inthe z-axis has several advantages. For example, this embodiment savesthe cost of providing a separate inertial mass and saves space and totalweight in the device, which are important considerations in the homeconsumer market. Another advantage of the actuator assembly 80 is thatit has a very low profile in the z-axis dimension. This is allowed bythe orientation of the actuator 66 in the x-y plane, e.g. the axis ofrotation A of the actuator shaft 84 is parallel to the z-axis. Thismakes the actuator assembly 80 very suitable for use in low-profilehousings.

[0061] In some embodiments, a larger actuator 66 can be used to bothoutput greater magnitude forces and to act as a larger inertial mass,resulting in higher magnitude haptic sensations as experienced by theuser. Or, an additional mass can be coupled to the actuator 66 shown inthe embodiment of FIG. 3 to provide a larger mass and overallhigher-magnitude haptic sensations. When tuning the system for suchforces, the resonant frequency of the system should remain the same(e.g. 25 Hz is one tested frequency). Thus, the stiffness of the flexure68 may have to be modified to maintain the desired resonant frequencywhen increasing the size of the inertial mass. Members of the flexurecan be stiffened by increasing their width or by providing a stiffermaterial.

[0062] Of course, in other embodiments, the actuator need not be used asthe inertial mass. For example, copending provisional application No.60/______, entitled “Haptic Interface Device Providing Linear TactileSensations Using A Rotary Actuator,” filed Dec. 21, 1999, andincorporated herein by reference, discloses an actuator coupled to aflexure that provides a centering spring bias to a separate inertialmass coupled to the flexure, or an inertial mass that is incorporated aspart of the flexure.

[0063]FIG. 4 is a side elevational view illustrating another embodiment200 of the present invention, in which contact forces are applied to theuser. In general, a moving element is provided on the surface of thehousing and is moved by the actuator 18. The user contacts the movingelement with a portion of his or her hand and thus directly feels themotion of the element.

[0064] In the example of FIG. 4, actuator 18 has a moving portion 148which moves along the z-axis as described above with reference to FIG.2. The moving portion 148 is coupled to a link member 150, which iscoupled to a moveable cover portion 152 at its other end. The linkmember can be rotatably coupled to the actuator and rotatably coupled tothe cover portion 152 by mechanical bearings or other types ofcouplings, such as flex joints.

[0065] The cover portion 152 is preferably the same material as thehousing 13 and is preferably movably coupled to the housing 13. Forexample, a mechanical hinge 154 can be used to provide a rotationalcoupling between cover portion 152 and housing 13. Alternatively, aflexure or other moveable coupling can be used to allow rotational orlinear motion of the cover portion. The cover portion 152 can also bemade of a flexible material that can flex to provide its motion andcontact forces to the user, such as a rubber diaphragm.

[0066] The approximate linear motion of the actuator's moving portion148 can be used to drive the cover portion 150. Linear forces from theactuator 18 move the link member 150 and in turn move the cover portion150 approximately along the Z-axis. Although the cover portion 150actually rotates about the hinge in the embodiment of FIG. 4, the rangeof motion is preferably small enough to approximate linear motion.Preferably, the cover portion 150 has an origin position (rest position)in the middle. of its range of motion so that the actuator 18 can moveit both up and down. Also, a centering spring bias is preferablyprovided to move the cover portion to the origin position when no forceis applied by the actuator (and by the user). These embodiment isdescribed in greater detail in copending patent application Ser. No.09/103,281, incorporated herein by reference.

[0067] In other embodiments, different moving elements can be actuatedto provide contact forces. For example, a button 16 can be coupled to alink member 150 or more directly to a moving portion of actuator 18. Thebutton 16 can be moved in its degree of freedom by the actuator toprovide contact forces to a user who is contacting the button. As withthe cover portion 150, the button is preferably centered in its range ofmotion by a centering spring bias provided by a physical spring orcompliance in the button. This embodiment is described in greater detailin copending patent application Ser. No. 09/253,132, incorporated hereinby reference.

[0068] Like the trackball device providing inertial forces, the actuatorsystem providing contact forces can be tuned to amplify output forces.Feet 60 can be made compliance, and compliance can also be used in theactuator's moving member 148, the link member 150, and the movingelement itself, where appropriate.

[0069] Of course, both the inertial forces described with reference toFIGS. 2 and 3 as well as the contact forces of FIG. 4 can be included ina single embodiment. For example, the link member 150 and moving element(cover portion, button, or other moving member) can b coupled to themoving inertial mass. Such an embodiment advantageously providesinertial forces that can always be felt by the user, regardless of howthe housing is contacted, as well as contact forces which can becompelling in particular situations.

[0070]FIG. 5 is a block diagram illustrating one embodiment of the forcefeedback system of the present invention including a localmicroprocessor and a host computer system.

[0071] Host computer system 14 preferably includes a host microprocessor200, a clock 202, a display screen 26, and an audio output device 204.The host computer also includes other well known components, such asrandom access memory (RAM), read-only memory (ROM), and input/output(I/O) electronics (not shown). Display screen 26 displays images of agame environment, operating system application, simulation, etc. Audiooutput device 204, such as speakers, is preferably coupled to hostmicroprocessor 200 via amplifiers, filters, and other circuitry wellknown to those skilled in the art and provides sound output to user whenan “audio event” occurs during the implementation of the hostapplication program. Other types of peripherals can also be coupled tohost processor 200, such as storage devices (hard disk drive, CD ROMdrive, floppy disk drive, etc.), printers, and other input and outputdevices.

[0072] Trackball device 12 is coupled to host computer system 14 by abi-directional bus 20 The bi-directional bus sends signals in eitherdirection between host computer system 14 and the interface device. Bus20 can be a serial interface bus, such as an RS232 serial interface,RS-422, Universal Serial Bus (USB), MIDI, or other protocols well knownto those skilled in the art; or a parallel bus or wireless link. Forexample, the USB standard provides a relatively high speed interfacethat can also provide power to actuator 18.

[0073] Device 12 can include a local microprocessor 210. Localmicroprocessor 210 can optionally be included within the housing ofdevice 12 to allow efficient communication with other components of thedevice. Processor 210 is considered local to device 12, where “local”herein refers to processor 210 being a separate microprocessor from anyprocessors in host computer system 14. “Local” also preferably refers toprocessor 210 being dedicated to haptic feedback and sensor I/O ofdevice 12. Microprocessor 210 can be provided with software instructionsto wait for commands or requests from computer host 14, decode thecommand or request, and handle/control input and output signalsaccording to the command or request. In addition, processor 210 canoperate independently of host computer 14 by reading sensor signals andcalculating appropriate forces from those sensor signals, time signals,and stored or relayed instructions selected in accordance with a hostcommand. Some examples of microprocessors that can be used as localmicroprocessor 210 include the MC68HC711E9 by Motorola, the PIC16C74 byMicrochip, and the 82930AX by Intel Corp., for example, as well as moresophisticated force feedback processors such as the Immersion TouchsenseProcessor from Immersion Corp. Microprocessor 210 can include onemicroprocessor chip, multiple processors and/or co-processor chips,and/or digital signal processor (DSP) capability.

[0074] Microprocessor 210 can receive signals from sensor 212 andprovide signals to actuator 18 in accordance with instructions providedby host computer 14 over bus 20. For example, in a local controlembodiment, host computer 14 provides high level supervisory commands tomicroprocessor 210 over bus 20, and microprocessor 210 decodes thecommands and manages low level force control loops to sensors and theactuator in accordance with the high level commands and independently ofthe host computer 14. This operation is described in greater detail inU.S. Pat. Nos. 5,739,811 and 5,734,373, both incorporated by referenceherein. In the host control loop, force commands are output from thehost computer to microprocessor 210 and instruct the microprocessor tooutput a force or force sensation having specified characteristics. Thelocal microprocessor 210 reports data to the host computer, such aslocative data that describes the position of the sphere 15 in one ormore provided degrees of freedom. The data can also describe the statesof buttons 16 and safety switch 232. The host computer uses the data toupdate executed programs. In the local control loop, actuator signalsare provided from the microprocessor 210 to actuator 18 and sensorsignals are provided from the sensor 212 and other input devices 218 tothe microprocessor 210. Herein, the term “tactile sensation” refers toeither a single force or a sequence of forces output by the actuator 18which provide a sensation to the user. For example, vibrations, a singlejolt, or a texture sensation are all considered tactile sensations. Themicroprocessor 210 can process inputted sensor signals to determineappropriate output actuator signals by following stored instructions.The microprocessor may use sensor signals in the local determination offorces to be output on the user object, as well as reporting locativedata derived from the sensor signals to the host computer.

[0075] In yet other embodiments, other hardware can be provided locallyto device 12 to provide functionality similar to microprocessor 210. Forexample, a hardware state machine incorporating fixed logic can be usedto provide signals to the actuator 18 and receive sensor signals fromsensors 212, and to output tactile signals according to a predefinedsequence, algorithm, or process. Techniques for implementing logic withdesired functions in hardware are well known to those skilled in theart. Such hardware can be better suited to less complex force feedbackdevices, such as the device of the present invention.

[0076] In a different, host-controlled embodiment, host computer 14 canprovide low-level force commands over bus 20, which are directlytransmitted to the actuator 18 via microprocessor 210 or othercircuitry. Host computer 14 thus directly controls and processes allsignals to and from the device 12, e.g. the host computer directlycontrols the forces output by actuator 18 and directly receives sensorsignals from sensor 212 and input devices 218. This embodiment may bedesirable to reduce the cost of the force feedback device yet further,since no complex local microprocessor 210 or other processing circuitryneed be included in the device. Furthermore, since one actuator 18 isused with forces not provided in the primary sensed degrees of freedom,the local control of forces by microprocessor 210 may not be necessaryin the present invention to provide the desired quality of forces.

[0077] In the simplest host control embodiment, the signal from the hostto the device can be a single bit that indicates whether to pulse theactuator at a predefined frequency and magnitude. In a more complexembodiment, the signal from the host could include a magnitude, givingthe strength of the desired pulse. In yet a more complex embodiment, thesignal can include a direction, giving both a magnitude and a sense forthe pulse. In still a more complex embodiment, a local processor can beused to receive a simple command from the host that indicates a desiredforce value to apply over time. The local microprocessor then outputsthe force value for the specified time period based on the one command,thereby reducing the communication load that must pass between host anddevice. In an even more complex embodiment, a high-level command withtactile sensation parameters can be passed to the local processor on thedevice which can then apply the full sensation independent of hostintervention. Such an embodiment allows for the greatest reduction ofcommunication load. Finally, a combination of numerous methods describedabove can be used for a single device 12.

[0078] Local memory 222, such as RAM and/or ROM, is preferably coupledto microprocessor 210 in device 12 to store instructions formicroprocessor 210 and store temporary and other data. For example,force profiles can be stored in memory 222, such as a sequence of storedforce values that can be output by the microprocessor, or a look-uptable of force values to be output based on the current position of theuser object. In addition, a local clock 224 can be coupled to themicroprocessor 210 to provide timing data, similar to the system clockof host computer 14; the timing data might be required, for example, tocompute forces output by actuator 18 (e.g., forces dependent oncalculated velocities or other time dependent factors). In embodimentsusing the USB communication interface, timing data for microprocessor210 can be alternatively retrieved from the USB signal.

[0079] For example, host computer 14 can send a “spatial representation”to the local microprocessor 210, which is data describing the locationsof some or all the graphical objects displayed in a GUI or othergraphical environment which are associated with forces and thetypes/characteristics of these graphical objects. The microprocessor canstore such a spatial representation in local memory 222, and thus willbe able to determine interactions between the user object and graphicalobjects (such as the rigid surface) independently of the host computer.In addition, the microprocessor can be provided with the necessaryinstructions or data to check sensor readings, determine cursor andtarget positions, and determine output forces independently of hostcomputer 18. The host could implement program functions (such asdisplaying images) when appropriate, and synchronization commands can becommunicated between the microprocessor and host 18 to correlate themicroprocessor and host processes. Also, the local memory can storepredetermined force sensations for the microprocessor that are to beassociated with particular types of graphical objects. Alternatively,the computer 14 can directly send force feedback signals to the device12 to generate tactile sensations.

[0080] Sensors 212 sense the position or motion of the device (e.g. thesphere 15) in its degrees of freedom and provides signals tomicroprocessor 210 (or host 14) including information representative ofthe position or motion. Sensors suitable for detecting motion of atrackball sphere include digital optical encoders frictionally coupledto the sphere, as is well known to those skilled in the art. Opticalsensor systems, linear optical encoders, potentiometers, opticalsensors, velocity sensors, acceleration sensors, strain gauge, or othertypes of sensors can also be used, and either relative or absolutesensors can be provided. Optional sensor interface 214 can be used toconvert sensor signals to signals that can be interpreted by themicroprocessor 210 and/or host computer system 14, as is well known tothose skilled in the art.

[0081] Actuator 18 transmits forces to the housing 13 of the device 12as described above with reference to FIG. 2 in response to signalsreceived from microprocessor 210 and/or host computer 14. Actuator 18can be a linear or rotary voice coil motor, linear or rotary DC motor,solenoid, pager motor, moving magnet actuator, piezo-electric actuator,etc. Actuator 18 is provided to generate inertial forces by moving aninertial mass; in the preferred embodiment, the mass is moved linearlyand approximately perpendicular to the surface on which the device issupported, and thus the actuator 18 does not generate force in thedegrees of freedom of motion of the sphere. Actuator 18 instead provides“informative” or “effect” forces that do not resist or assist motion.Actuator 18 can additionally or alternatively drive a moving element toprovide contact forces as described above. The sensors 212 detect theposition/motion of the device 12 in its planar degrees of freedom, andthis sensing is not substantially affected by the output of forces byactuator 18.

[0082] The actuator described herein has the ability to apply shortduration force sensation on the housing of the device (and/or on theuser's hand). This short duration force sensation is described herein asa “pulse.” Ideally the “pulse” is directed substantially along a Z axisorthogonal to the X-Y plane of the support surface 22. In progressivelymore advanced embodiments, the magnitude of the “pulse” can becontrolled; the sense of the “pulse” can be controlled, either positiveor negative biased; a “periodic force sensation” can be applied on thehousing, where the periodic sensation can have a magnitude and afrequency, e.g. a sine wave; the periodic sensation can be selectableamong a sine wave, square wave, saw-toothed-up wave, saw-toothed-down,and triangle wave; an envelope can be applied to the period signal,allowing for variation in magnitude over time; and the resulting forcesignal can be “impulse wave shaped” as described in U.S. Pat. No.5,959,613. There are two ways the period sensations can be communicatedfrom the host to the device. The wave forms can be “streamed” asdescribed in U.S. Pat. No. 5,959,613 and pending provisional patentapplication 60/160,401, both incorporated herein by reference. Or thewaveforms can be conveyed through high level commands that includeparameters such as magnitude, frequency, and duration, as described inU.S. Pat. No. 5,734,373.

[0083] Alternate embodiments can employ additional actuators forproviding tactile sensations or forces in the planar degrees of freedomof the device 12. For example, the device 12 can be enhanced with asecondary actuator that outputs forces on the sphere 15 to resist and/orassist motion of the sphere. For example, frictional rollers coupled tothe sphere can be driven by actuators, as described in U.S. Pat. Nos.5,623,582 and 5,889,670, both incorporated herein by reference. In someembodiments, because of power constraints, this second actuator can bepassive (i.e., it dissipates energy). The passive actuator can be abrake, such as a brake employing a very low power substrate such as amagneto-rheological fluid. Alternatively it could be a more traditionalmagnetic brake. The sphere can be rotated freely so long as the passivebrake is not engaged. When the brake is engaged, the user can feel thepassive resistance to motion of the sphere (in one and/or two degrees offreedom). The passive resistance can allow additional feel sensationsthat supplement the “pulse” and “vibration” sensations (described withreference to FIG. 6).

[0084] In yet other embodiments, an actuator can be provided to outputthe tactile feedback (such as pulses and vibrations) to the sphere 15itself instead of or in addition to the tactile feedback applied to thehousing 13. For example, a linear or rotary actuator can output pulseson the sphere 15 by vibrating a cylindrical roller in contact with thesphere. Or, a moving portion of an actuator can directly impact thesphere. However, such tactile sensations on the sphere may causeinaccurate cursor control or input for the user, which is generallyundesirable. A selective disturbance filter, as described in copendingapplication Ser. No. 08/839,249, can be used to filter out the forcedisturbances on the cursor control. However, in some embodiments thismay not be adequate since vibrations on the sphere are difficult tosense with accuracy and therefore difficult to filter. Other embodimentsmay provide accurate enough sensors, such as multiple emitter/detectorpairs sensing small motions of the sphere, which can allow for adequatedisturbance filtering.

[0085] Actuator interface 216 can be optionally connected betweenactuator 18 and microprocessor 110 to convert signals frommicroprocessor 210 into signals appropriate to drive actuator 18.Interface 38 can include power amplifiers, switches, digital to analogcontrollers (DACs), analog to digital controllers (ADCs), and othercomponents, as is well known to those skilled in the art.

[0086] Other input devices 218 are included in device 12 and send inputsignals to microprocessor 210 or to host 14 when manipulated by theuser. Such input devices include buttons 16 and can include additionalbuttons, dials, joysticks, switches, scroll wheels, or other controls ormechanisms. These other input devices 218 can be positioned on thehousing 13 in some embodiments.

[0087] Power supply 220 can optionally be included in device 12 coupledto actuator interface 216 and/or actuator 18 to provide electrical powerto the actuator. or be provided as a separate component. Alternatively,and more preferably, power can be drawn from a power supply separatefrom device 12, or power can be received across a USB or other bus.Also, received power can be stored and regulated by device 12 and thusused when needed to drive actuator 18 or used in a supplementaryfashion. Because of the limited power supply capabilities of USB, apower storage device may be required in the device to ensure that peakforces can be applied (as described in U.S. Pat. No. 5,929,607,incorporated herein by reference). For example, power can be stored overtime in a capacitor or battery and then immediately dissipated toprovide a jolt sensation to the device. Alternatively, this technologycan be employed in a wireless device 12, in which case battery power isused to drive the tactile actuator. In one embodiment, the battery canbe charged by an electric generator on board the device 12, thegenerator driven by the user's motions of the device. For example, thesphere 15 can turn a frictional roller or shaft that is coupled to andrecharges the generator.

[0088] A safety switch 232 can optionally be included to allow a user todeactivate actuator 18 for safety reasons. For example, the user mustcontinually activate or close safety switch 232 during operation ofdevice 12 to enable the actuator 18. If, at any time, the safety switchis deactivated (opened), power from power supply 220 is cut to actuator18 (or the actuator is otherwise disabled) as long as the safety switchis opened. Embodiments include an optical switch, an electrostaticcontact switch, a button or trigger, a hand weight safety switch, etc.

[0089]FIG. 6 is a diagram of display screen 26 of host computer 14showing a graphical user interface for use with the present invention,which is one type of graphical environment with. which the user caninteract using the device of the present invention. The haptic feedbacktrackball device 12 of the present invention can provide tactilesensations that make interaction with graphical objects more compellingand more intuitive. The user typically controls a cursor 246 to selectand manipulate graphical objects and information in the graphical userinterface. The cursor is moved according to a position control paradigm,where the position of the cursor corresponds to a position of the spherein its rotational workspace. Windows 250 and 252 display informationfrom application programs running on the host computer 14. Menu elements256 of a menu 254 can be selected by the user after a menu heading orbutton such as start button 255 is selected. Icons 256, 260, and 261 andweb links 262 are displayed features that can also be selected. Tactilesensations associated with these graphical objects can be output usingactuator 18 based on signals output from the local microprocessor orhost computer.

[0090] A basic tactile functionality desired for the device describedherein is a “pulse” (or jolt) sensation that is output when the cursoris (a) moved between menu elements 256 of a menu 254, (b) moved on to anicon 256, button, hyperlink 262, or other graphical target, (c) movedacross a boundary of a window 250 or 252, (d) moved overapplication-specific elements in a software title such as nodes in aflow chart, the points of a drawing, or the cells of a spread sheet. Theappropriate sensation for this simple cursor interaction is a quick,abrupt “pulse” or “pop.” This can be achieved by applying a crisp, shortforce between the inertial mass and the housing of the device, e.g. bymoving the inertial mass in one or a small number of oscillations. Forexample, a pulse can include a single impulse of force that quicklyrises to a desired magnitude and then is turned off or quickly decaysback to zero or small magnitude.

[0091] A vibration can also be output, which can include a series ofpulses applied periodically over a particular time period at aparticular frequency. The time-varying force can be output according toa force vs. time waveform that is shaped like a sine wave, trianglewave, sawtooth wave, or other shape of wave. The vibration is caused byan inertial mass and/or moving contact element oscillating back andforth.

[0092] In some embodiments, the sensation of a “spatial texture” may beoutput by correlating pulses and/or vibrations with the motion of thecursor over a graphical object or area. This type of force can depend onthe position of the sphere 15 in its workspace (or on the position ofthe cursor in the graphical user interface). For example, the cursor canbe dragged over a graphical grating and pulses can be correlated withthe spacing of the grating. Thus, texture bumps are output depending onwhether the cursor has moved over the location of a bump in a graphicalobject; when the sphere is positioned between “bumps” of the texture, noforce is output, and when the sphere moves over a bump, a force isoutput. This can be achieved by host control (e.g., the host sends thepulses as the cursor is dragged over the grating) or by local control(e.g., the host sends a high level command with texture parameters andthe sensation is directly controlled by the device). In other cases atexture can be performed by presenting a vibration to a user, thevibration being dependent upon the current velocity of the sphere in itsworkspace. When the sphere is stationary, the vibration is deactivated;as the sphere moves faster, the frequency and magnitude of the vibrationis increased. This sensation could be controlled locally by the deviceprocessor, or be controlled by the host. Local control by the device mayeliminate communication burden in some embodiments. Other spatial forcesensations can also be output. In addition, any of the described forcesensations herein can be output by actuator 18 simultaneously orotherwise combined as desired.

[0093] The host computer 14 can coordinate tactile sensations withinteractions or events occurring within the host application. Theindividual menu elements 256 in the menu can be associated with forces.In one interaction, when the cursor is moved across menu elements 256 inmenu 254 of the graphical user interface, “pulse” sensations areapplied. The sensations for certain menu choices can be stronger thanothers to indicate importance or frequency of use, i.e., the most usedmenu choices can be associated with higher-magnitude (stronger) pulsesthan the less used menu choices. Also, disabled menu choices can have aweaker pulse, or no pulse, to indicate that the menu choice is notenabled at that time. Furthermore, when providing tiled menus in which asub-menu is displayed after a particular menu element is selected, as inMicrosoft Windows¹⁹⁸, pulse sensations can be sent when a sub-menu isdisplayed. This can be very useful because users may not expect asub-menu to be displayed when moving a cursor on a menu element.

[0094] Pulse sensations can also be output based on interaction betweencursor 246 and a window. For example, a pulse can be output when thecursor is moved over a border of a window 250 or 252 to signal the userof the location of the cursor. When the cursor 246 is moved within thewindow's borders, a texture force sensation can be output. The texturecan be a series of bumps that are spatially arranged within the area ofthe window in a predefined pattern; when the cursor moves over adesignated bump area, a pulse sensation is output when the cursor movesover designated pulse points or lines. A pulse can also be output whenthe cursor is moved over a selectable object, such as a link 254 in adisplayed web page or an icon 256. A vibration can also be output tosignify a graphical object which the cursor is currently positionedover. Furthermore, features of a document displaying in window 250 or252 can also be associated with force sensations.

[0095] In another interaction, when the cursor is moved over an icon256, folder, hyperlink 262, or other graphical target, a pulse sensationis applied. The sensation associated with some elements can be strongerthan others to indicate importance or just to differentiate differentelements. For example, icons can be associated with stronger pulses thanfolders, where the folders can be associated with stronger pulses thantool bar items. Also, the strength of a pulse can be associated with thedisplayed size of the graphical element, where a large tool bar icon canbe associated a stronger pulse than a small tool bar icon. On web pagesthis is particularly interesting, where small graphical targets can beassociated with weaker pulses than large graphical targets. Also, on webpages check boxes and hyperlinks can feel different than buttons orgraphical elements based on pulse strength. The magnitude of the pulsescan also depend on other characteristics of graphical objects, such asan active window as distinguished from a background window, file foldericons of different priorities designated by the user, icons for games asdistinguished from icons for business applications, different menu itemsin a drop-down menu, etc. Methods of adding tactile sensations to webpages is described in U.S Pat. No. 5,956,484 and co-pending patentapplication Ser. No. 08/571,606, both incorporated herein by reference.

[0096] In another interaction, when a document is being scrolled, apulse sensation can be used to indicate the passing of page breaks orother demarcations, e.g. when a particular area or feature of a scrolledpage is scrolled past a particular area of the window. In a relatedtactile sensations, when a document is being scrolled, a vibrationsensation can be used to indicate the motion. The frequency of thevibration can be used to indicate the speed of the scrolling, where fastscrolling is correlated with higher-frequency sensations than slowscrolling.

[0097] In other related scrolling interactions, when a down-arrow ispressed on a scroll bar, a vibration can be displayed on the device toindicate that scrolling is in process. When using a graphical slider andreaching the end of the slider's travel, a pulse can be used to indicatethat the end of travel has been reached. When using a slider bar thathas “tick marks”, pulse sensations can be used to indicate the locationof the “ticks.” In some slider bars there is only a single tick mark toindicate the center of the slider bar; a pulse can be output to informthe user when center is reached. In other slider bars there are ticks ofdifferent size (for example the center tick may be more important thanthe others). In such an embodiment, different strength pulses can beused, larger strength indicating the more important ticks. For example,when setting the balance on system audio speakers, a slider is used withtick marks. The user can feel the ticks with the present invention byproviding associated pulses, especially the center tick which indicatescenter balance. Pulses can also be provided for volume controls. Inother instances, strength of a vibration can be correlated with theadjustment of a volume control to indicate magnitude. In yet otherinstances the frequency of a vibration can be correlated with theadjustment of a volume control to indicate magnitude.

[0098] In other interactions, when dragging a graphical object in agraphical user interface, such as an icon, or stretching an element suchas a line, a vibration sensation can be used to indicate that thefunction is active.

[0099] In some cases a user performs a function, like cutting or pastinga document, and there is a delay between the button press that commandsthe function and the execution of the function, due to processing delaysor other delays. A pulse sensation can be used to indicate that thefunction (the cut or paste) has been executed.

[0100] Tactile sensations can also be associated with particular eventsthat the user may or may not have control over. For example, when emailarrives or an appointment reminder is displayed, a pulse or a vibrationcan be output to notify the user of the event. This is particularlyuseful for disabled users (e.g., blind or deaf users). When an errormessage or other system event is displayed in a dialog box on the hostcomputer, a pulse or vibration can be used to draw the user's attentionto that system event. When the host system is “thinking,” requiring theuser to wait while a function is being performed or accessed (usuallywhen a timer is displayed by the host) it is often a surprise when thefunction is complete. If the user takes his or her eyes off the screen,he or she may not be aware that the function is complete. A pulsesensation can be sent to indicate that the “thinking” is over. Thetactile sensations can be varied to signify different types of events ordifferent events of the same type. For example, vibrations of differentfrequency can each be used to differentiate different events ordifferent characteristics of events, such as particular users sendingemail, the priority of an event, or the initiation or conclusion ofparticular tasks (e.g. the downloading of a document or data over anetwork).

[0101] Many tactile sensations can be coordinated with interactions andevents occurring within specific types of applications. For example, ina gaming application, a wide variety of periodic sensations can be usedto enhance various gaming actions and events, such as engine vibrations,weapon fire, crashes and bumps, rough roads, explosions, etc. Thesesensations can be implemented as button reflexes as described in U.S.Pat. No. 5,691,898, incorporated herein by reference.

[0102] In a spread sheet application, pulse sensations can be used toindicate when the cursor is moved from one element or cell to another.Stronger pulses can be used to indicate when a particular or predefinedrow, column, or cell is encountered. Ideally the user who is craftingthe spreadsheet can define the strength of the sensation as part of thespreadsheet construction process as well as the particular featuresassigned to particular pulse strengths.

[0103] In a word processor, pulse sensations can be output to allow theuser to feel the boundaries between words, the spaces between words, thespaces between lines, punctuation, highlights, bold text, or othernotable elements. When adjusting the tab spacing in a word processor,pulses can be used to indicate the adjustment of the graphical tabmarkers. Stronger pulses can be used on the spaces at certain multiples.When writing an outline in a word processor in which a hierarchy ofparagraphs is imposed, pulses can be used to indicate when the cursor ison a particular outline line of a given hierarchy.

[0104] In a drawing application that allows a user to lay down colorpixels using a “spray can” metaphor, a vibration can be output duringthe “spraying” process to make the spray-can metaphor more compelling tothe user. Drawing or CAD programs also have many other features whichcan be associated with pulses or other sensations, such as displayed (orinvisible) grid lines or dots, control points of a drawn object,outlines or borders of objects, etc.

[0105] On web pages, pulse or vibration content can be used to enhancethe user experience, e.g. for web objects such as web page links, entrytext boxes, graphical buttons, and images. Methods of adding suchcontent are described in U.S. Pat. No. 5,956,484 and co-pending patentapplication Ser. No. 08/571,606, both incorporated herein by reference.

[0106] There may be certain cases where a user might want to be able toturn on or turn off the pulse feedback for a particular feature. Forexample, when adding a letter to a word in a word processor it is usefulto be able to feel the letters as pulses as the cursor is moved fromletter to letter along a word. However, this sensation is not alwaysdesired by the user. Therefore the sensation can preferably be enabledor disabled by a software selector such as a check box, and/or byhardware such as pressing a button on the device 12. In other cases orembodiments, a feature can be enabled or disabled depending upon thevelocity at which the sphere 15 is being moved. For example, if the useris moving the cursor very quickly across the displayed desktop, the useris probably not trying to select a graphical object in the path of thecursor. In that case the pulses could be a distraction as the cursorpasses over icons or over window borders. Therefore, it would beadvantageous if the host software (or the software/firmware run by alocal microprocessor) attenuated or eliminated the pulses when moving ator greater than a threshold velocity. Conversely, when the user ismoving the cursor slowly he or she is likely trying to select or engagea graphical target; in that case the pulses could be active or evenaccentuated with a higher magnitude.

[0107] A software designer may want to allow a user to access a softwarefunction by positioning the cursor over an area on the screen, but notrequire pressing a button on the device (as is the typical way toexecute a function, often called “clicking”). Currently, it isproblematic to allow “click-less” execution because a user has physicalconfirmation of execution when pressing a button. A pulse sent to thetactile device of the present invention can act as that physicalconfirmation without the user having to press a button. For example, auser can position a cursor over a web page element, and once the cursoris within the desired region for a given period of time, an associatedfunction can be executed. This is indicated to the user through atactile pulse sent to the device.

[0108] If additional actuator(s) are being used to supplement theprimary actuator 18, such as an actuator (e.g., a low-power brake) forproviding forces on the sphere 15 as described with respect to FIG. 5,then the forces provided by the additional actuator(s) can allowadditional feel sensations that supplement the “pulse” and “vibration”sensations described above. For example, when a user drags an icon withthe sphere, a passive resistance force from a brake output on the spherecan provide a dragging (damping) sensation to the user. The larger theobject to be dragged (in displayed size or other measurablecharacteristic), the more resistance is applied. Also, when a userstretches an image, the passive resistance force can provide a draggingsensation. The larger the object to be dragged, the more resistance isapplied. The use of both active and passive haptic feedback can be usedsynergistically; for example, passive resistance can be useful to slowdown sphere movement when selecting menu items, but since passivefeedback can only be output when the sphere is being moved by the user,active feedback is useful to be output when the sphere is at rest ormoving slowly. An embodiment employing passive braking can also employthe “desired play” methodology described in U.S. Pat. No. 5,767,839,incorporated herein by reference, to achieve enhanced functionality.

[0109] While this invention has been described in terms of severalpreferred embodiments, it is contemplated that alterations, permutationsand equivalents thereof will become apparent to those skilled in the artupon a reading of the specification and study of the drawings. Forexample, many. different types of tactile sensations can be providedwith the actuator of the present invention and many different types ofactuators can be used. Furthermore, certain terminology has been usedfor the purposes of descriptive clarity, and not to limit the presentinvention. It is therefore intended.that the following appended claimsinclude alterations, permutations, and equivalents as fall within thetrue spirit and scope of the present invention.

What is claimed is:
 1. A haptic feedback trackball device coupled to ahost computer implementing a host application program, said hapticfeedback device used by a user to provide input to said host computer,the haptic feedback device comprising: a housing that is physicallycontacted by said user, said housing resting on a support surface; asphere positioned in said housing, said sphere rotatable in two rotarydegrees of freedom; a sensor device coupled to said housing of saidmouse, said sensor device detecting said movement of said sphere in saidrotary degrees of freedom and outputting sensor signals representativeof said movement; and an actuator coupled to said housing of saiddevice, said actuator operative to apply a force to said housingapproximately along an axis that is substantially perpendicular to saidsupport surface, wherein said force is transmitted to said usercontacting said housing.
 2. A trackball device as recited in claim 1wherein said force is an inertial force that is output approximatelyalong said axis that is substantially perpendicular to said supportsurface, wherein said actuator outputs said inertial force to saidhousing by moving an inertial mass.
 3. A trackball device as recited inclaim 2 wherein said inertial force is correlated with a graphicalrepresentation displayed by said host computer, wherein a position ofsaid sphere in said rotary degrees of freedom corresponds with aposition of a cursor displayed in said graphical representation.
 4. Atrackball device as recited in claim 2 wherein said inertial force is apulse correlated with the interaction of a user-controlled cursor with agraphical object displayed in a graphical user interface.
 5. A trackballdevice as recited in claim 4 wherein said pulse is output when saidcursor moves between menu items in a displayed graphical menu.
 6. Atrackball device as recited in claim 1 wherein said force is included ina force sensation, said force sensation being one of a pulse, vibration,and texture force.
 7. A trackball device as recited in claim 1 whereinsaid force is a contact force that is provided by driving a movingelement that contacts said user using said actuator.
 8. A trackballdevice as recited in claim 7 wherein said moving element is a coverportion of said housing, said cover portion being movably coupled to abase portion of said housing.
 9. A trackball device as recited in claim7 wherein said moving element is a button, said button also forproviding button input to said host computer.
 10. A trackball device asrecited in claim 1 wherein at least one compliant element is coupled tosaid housing, said compliant element supporting said housing on saidsupport surface and including a compliance, said compliance allowingsaid force to be greater in magnitude than if said housing contactedsaid support surface directly.
 11. A trackball device as recited inclaim 10 wherein said at least one compliant element comprises rubber orfoam.
 12. A trackball device as recited in claim 1 further comprising amicroprocessor, separate from said host computer, coupled to said sensordevice and to said actuator, said microprocessor operative to receivehost commands from said host computer and output force signals to saidactuator for controlling said force, and operative to receive saidsensor signals from said sensor device, process said sensor signals, andreport locative data to said host computer derived from said sensorsignals and indicative of said movement of said sphere.
 13. A trackballdevice as recited in claim 1 wherein said actuator outputs said force inresponse to a command received by said trackball device from said hostcomputer.
 14. A haptic feedback trackball device coupled to a hostcomputer implementing a host application program, said haptic feedbackdevice used by a user to provide input to said host computer, the hapticfeedback device comprising: a housing that is physically contacted bysaid user, said housing resting on a support surface; a spherepositioned in said housing, said sphere rotatable in two rotary degreesof freedom; a sensor device coupled to said housing of said mouse, saidsensor device detecting said movement of said sphere in said rotarydegrees of freedom and outputting sensor signals representative of saidmovement; an actuator coupled to said housing of said device, saidactuator operative to apply a force to said housing, wherein said forceis transmitted to said user contacting said housing; and at least onecompliant element coupled between a portion of said housing contacted bysaid user and said support surface, said at least one compliant elementamplifying said force output from said actuator by allowing saidcontacted portion of said housing to move with respect to said supportsurface.
 15. A trackball device as recited in claim 14 wherein said atleast one compliant element includes at least one compliant footprovided between said housing and said support surface.
 16. A trackballdevice as recited in claim 14 wherein said at least one compliantelement includes a compliant coupling provided between said contactedportion of said housing and a different portion of said housing.
 17. Atrackball device as recited in claim 14 wherein said force is outputapproximately along an axis that is substantially perpendicular to saidsupport surface.
 18. A trackball device as recited in claim 14 whereinsaid force is correlated with the interaction of a user-controlledcursor with a graphical object displayed in said graphical environment.19. A trackball device as recited in claim 14 wherein said force is aninertial force, and wherein said actuator outputs said inertial force tosaid housing by moving an inertial mass.
 20. A trackball device asrecited in claim 19 wherein said actuator is provided in an actuatorassembly that includes a flexure, said flexure providing a centeringspring force to said inertial mass.
 21. A trackball device as recited inclaim 14 wherein said force is a contact force, and wherein saidactuator drives a moving element that contacts said user.
 22. Atrackball device as recited in claim 14 further comprising amicroprocessor, separate from said host computer, coupled to said sensorand to said actuator, said microprocessor operative to receive hostcommands from said host computer and output force signals to saidactuator for controlling said force, and operative to receive saidsensor signals from said sensors, process said sensor signals, andreport locative data to said host computer derived from said sensorsignals and indicative of said movement of said sphere.
 23. A trackballdevice as recited in claim 14 wherein said actuator is a first actuator,and further comprising a second actuator for outputting a force on saidsphere in one of said two degrees of freedom of said sphere.
 24. Atrackball device as recited in claim 23 wherein said second actuator isa passive brake for providing a resistance to motion of said sphere. 25.A method for providing haptic feedback to a user interacting with agraphical environment displayed by a host computer, the usermanipulating a trackball device, the method comprising: detecting themotion of a sphere of said trackball device in two rotary degrees offreedom and providing an indication of said motion to said hostcomputer; receiving information from said host computer indicating thata tactile sensation is to be output, said tactile sensation beingcorrelated with an interaction or event occurring within said graphicalenvironment; and outputting a force on the housing of said trackballdevice using an actuator, said force being approximately along an axissubstantially perpendicular to a surface supporting said trackballdevice, wherein said user can contact said housing and experience saidforce.
 26. A method as recited in claim 25 wherein said force is aninertial force caused by moving an inertial mass coupled to saidactuator.
 27. A method as recited in claim 25 wherein said force is acontact force caused by driving a moving element with said actuator tocontact said user.